Detecting associates

ABSTRACT

Detecting, for a content item, associated preference events is disclosed. For the content item, a plurality of preference events from a plurality of users is received. The received preference events are accumulated. Associated events are detected. The effect of the events is reduced when assigning a status to the item.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/283,139, filed Sep. 8, 2008 and entitled DETECTING ASSOCIATES, whichclaims priority to U.S. Provisional Patent Application No. 60/967,910,filed Sep. 6, 2007 and entitled DETECTING ASSOCIATES AND AUTOMATICALLYADAPTING THRESHOLDS, all of which are incorporated herein by referencefor all purposes.

BACKGROUND OF THE INVENTION

Popular content repositories, review and voting sites, and other socialcollaborative networks generally contain a vast amount of content. Suchsites typically encourage users to rate the content (includingphotographs, news stories and journal entries, videos, products andother items, and services, etc.) to aid other users in locatinginteresting or otherwise desirable content and avoiding content that maynot be of interest.

Unfortunately, unscrupulous individuals may attempt to leverage thepopularity of such sites for financial gain such as by directing trafficto advertising and other self promotional material (e.g. spamming). Theymay also attempt to unfairly wage “smear campaigns” against legitimatecontent, such as competitors' products. One way of engaging in either ofthese behaviors is to band together with other users (whether they areother individuals or additional, fictitious users under the control of asingle user) to rate content in a concerted manner for mutual benefit.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1A illustrates an embodiment of an environment for collecting andmanaging content contributions.

FIG. 1B illustrates an embodiment of an interface to a preferencesystem.

FIG. 2 is a flow chart illustrating an embodiment of a process forreceiving a story submission.

FIG. 3 is a flow chart illustrating an embodiment of a process forreceiving a story submission.

FIG. 4 illustrates an embodiment of a story permalink.

FIG. 5 illustrates an embodiment of an interface to a preference system.

FIG. 6 illustrates an embodiment of an interface to a preference system.

FIG. 7 is a flow chart illustrating an embodiment of a process forrecording a preference for a content contribution.

FIG. 8A illustrates an embodiment of a visualization interface.

FIG. 8B illustrates an embodiment of a visualization interface.

FIG. 8C illustrates an embodiment of a visualization interface.

FIG. 8D illustrates an embodiment of a visualization interface.

FIG. 9 illustrates an embodiment of a visualization interface.

FIG. 10 illustrates an embodiment of a visualization interface.

FIG. 11A illustrates an embodiment of a visualization interface.

FIG. 11B illustrates an embodiment of a visualization interface.

FIG. 11C illustrates an embodiment of a visualization interface.

FIG. 12A is an example of a content contribution.

FIG. 12B illustrates an embodiment of an interface to a preferencesystem.

FIG. 12C illustrates an embodiment of an interface to a preferencesystem.

FIG. 13A illustrates an embodiment of an interface to a preferencesystem.

FIG. 13B illustrates an embodiment of an interface to a preferencesystem.

FIG. 14 illustrates an embodiment of an interface to a preferencesystem.

FIG. 15 is a flow chart illustrating an embodiment of a process fordetecting associates.

FIG. 16 is a flow chart illustrating an embodiment of a process forpromoting content.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess, an apparatus, a system, a composition of matter, a computerreadable medium such as a computer readable storage medium or a computernetwork wherein program instructions are sent over optical orcommunication links. In this specification, these implementations, orany other form that the invention may take, may be referred to astechniques. A component such as a processor or a memory described asbeing configured to perform a task includes both a general componentthat is temporarily configured to perform the task at a given time or aspecific component that is manufactured to perform the task. In general,the order of the steps of disclosed processes may be altered within thescope of the invention.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

FIG. 1A illustrates an embodiment of an environment for collecting andmanaging content contributions. Users, such as user 104, submit content(hereinafter a “story contribution” and a “third party news articlecontribution”) to preference system 102. In the example shown, contentis submitted in part by providing to preference system 102 the uniformresource locator (URL) of a story, such as a story found on web page106.

Preference system 102 includes a web module 108 that provides typicalweb server functionality such as serving website 116, capturing userinput, and providing Really Simple Syndication (RSS) feed (110) support.In the example shown, web module 108 is an Apache HTTP server thatsupports running PHP scripts. Web module 108 is interfaced with adatabase 112, such as through a MySQL database backend.

As described in more detail below, users are made aware of the submittedcontent through website 116 and features such as RSS feeds. In additionto providing a link to the content (e.g., a hyperlink to web page 106)and information such as a summary of the story and the date and time itwas submitted, website 116 permits users to indicate their preferencesfor the content by making a variety of interactions. For example, userscan “digg” a story to indicate their like of its content, “bury” a storyto indicate problems with the content, and may also take other actionssuch as commenting on the content. These actions (including the initialsubmission of the content contribution) are referred to hereincollectively as “preference events.”

Whenever a preference event occurs (e.g., whenever a user submits,diggs, buries, or comments on content), the event is recorded indatabase 112 along with associated information such as the identity ofthe user and a time/date stamp. As described in more detail below,information recorded in database 112 is used in a variety of ways, suchas in conjunction with visualization tools that query database 112and/or make use of data extracted from database 112.

In some embodiments, a promotion engine 118 is included. As described inmore detail below, promotion engine 118 makes determinations such aswhat content should appear in various sections of website 116. Forexample, content that appears to be popular may be “promoted” bypromotion engine 118 by being displayed in a prominent location on themain page of website 116 for some period of time.

Cohort detection module 120 is configured receive information fromdatabase 112 and determines groups of cohorts, e.g. as a batch process.As described in more detail below, in some embodiments when content isevaluated for promotion (e.g., by the promotion engine), the promotionengine determines whether a threshold number of cohorts (e.g., at least4/8 of those individuals in a group) dugg the content. If so, it reducesthe impact of their actions, as appropriate.

In some embodiments, the infrastructure provided by portions ofpreference system 102 is located on and/or replicated across a pluralityof servers rather than the entirety of preference system 102 beingcollocated on a single platform. Such may be the case, for example, ifthe contents of database 112 are vast and/or there are many simultaneousvisitors to site 116.

FIG. 1B illustrates an embodiment of an interface to a preferencesystem. The example shown is an implementation of a portion of website116, as rendered in a browser. A user, known as “Alice” is logged intosite 116. Interface 150 includes a sidebar 152 that provides access tovarious system services. For example, by selecting region 154 of sidebar152, Alice is presented with an interface that permits her to view herprofile and manage account settings such as her current email addressand password; view previous preference events she's taken (her“history”); and access friend-related features described in more detailbelow. Region 156 provides an indication of whether Alice has anymessages and will present her with an interface to a message system(such as a mailbox) if selected. As described in more detail below, byselecting region 158, Alice will be presented with an interface throughwhich she can submit a news story for inclusion on system 102.

Region 160 displays a list of categories into which news stories aregrouped. If “View All” is selected, stories from all categories will bedisplayed in story window 164. As shown, the “Technology” category isselected. In some embodiments, visual indications of what category isselected are presented. In the example shown, the selected category ishighlight (represented here by stippling) at 166, and the title of thecategory appears above story window 164 at 168. In some embodiments,Alice can configure which topics are available to her on site 116. Forexample, if Alice dislikes Sports, she can configure interface 150 tonever show her any sports-related stories, even when viewing using the“View All” option.

Story window 164 typically includes one or more story entries 170. Inthe example shown, a story entry includes the title of a story, as wellas other associated information, such as who submitted the story andwhen, the external URL of the story, the category to which the storybelongs, and a summary of the story. As described in more detail below,links are provided to the story directly (such as by clicking on thetitle), as well as to an area of site 116 associated with the story,referred to herein as the story's permalink. For example, by clicking onthe comments link (176) of the story, Alice will be presented with thecomments portion of the permalink described in more detail below.

Story entry 170 also includes a problem reporting region 178. Users mayreport problems for a variety of reasons. For example, the first storyentry and the third story entry shown describe the same news—scientistssuperheating a gas. Alice has selected the problem, “duplicate” story,from problem reporting region 178. As described in more detail below,this is one form of burying a story. In some embodiments, buried storiesare displayed differently, or removed entirely from the user'sinterface. In the example shown, once a story is buried, it is greyedout, represented here by stippling (180).

As described in more detail below, each story has one or more scoresassociated with it. In the example shown, the “digg” score (172) foreach story is displayed, as is an interactive region beneath the score(box 174) that allows a user to “digg” the story. The first story hasbeen dugg 237 times, but has not been dugg by Alice. As described inmore detail below, if Alice were to select region 174, a variety ofactions would be immediately taken, including increasing the digg scoreof the story and updating the region's text from “digg it” to “dugg!” asshown in region 182.

Alice is currently viewing a “promoted stories” (184) view of storywindow 164. This means that all of the stories presented to Alice on thecurrent view of the interface have exceeded a promotion threshold. Oneexample of a promotion threshold is the raw number of diggs. Otherrequirements/factors may be used for thresholding in addition to orinstead of a digg score, such as requiring that a certain amount of timeelapse between story submission and story promotion, the speed withwhich stories are being dugg, information associated with users thathave dugg the story, etc. Because some threshold of users must agreethat a story has merit before being promoted, stories shown in promotedview 184 are unlikely to contain spam or otherwise be inherentlyinappropriate for Alice's viewing.

In some embodiments, different thresholds are used for differentstories, such as for stories in different categories. For example, thepromotion of a math related story may only require 100 diggs whereas astory about the president may require 500 diggs. Techniques forpromoting content are described more fully below.

If Alice selects the upcoming stories tab (186), only stories which havenot yet met the appropriate threshold will be displayed. For example,newly submitted stories which have not yet been “dugg” by a sufficientnumber of people will be presented by selecting tab 186. In someembodiments, if a story languishes in the upcoming stories pool for morethan a certain period of time without receiving a sufficient digg scoreto be promoted (e.g., for a week), the story is removed from the pooland can only be found via its permalink or through a search. In someembodiments, such stories are deleted from database 112. Such storiesare typically indicative of spam, inaccuracies, and old news. Similarly,if enough users bury a story, the story may be removed from the pooland/or database 112.

In other embodiments, other views of stories may be presented asapplicable, such as a view that unifies both the promoted and theupcoming stories. In the example shown, because Alice has selected the“Technology” category (166), only technology related stories arepresented in the promoted stories (184) and upcoming stories (186)views. Similarly, the topics of the presented stories (e.g., “Math,” areall subtopics of Technology). In some embodiments, the informationpresented with the story entry may vary, such as from topic to topic.For example, if Alice selected “View All” at 160, the listed topic maybe the top level category to which the story belongs (e.g.,“Technology”) or include a drilled down description (e.g., “WorldNews-Canada-Elections”).

As described in more detail below, portion 162 of interface 150 displaysthe recent activities (preference events) of Alice's friends. Forexample, in the last 48 hours, Alice's friends have submitted twostories, dugg twelve stories, and commented on sixteen stories, asreflected in dashboard 162. Of the twelve stories her friends have dugg,four of the stories have not yet been promoted. In some embodiments,assorted visual cues of her friends' activity are presented throughoutwebsite 116. In the example shown, stories dugg by Alice's friends arenotated by a banner (184) placed across the digg score. In other cases,other cues may be used, such as by changing the color of the story,and/or interactive behavior such as playing a sound or showing thefriend's avatar icon when, for example, Alice's cursor hovers over astory dugg by a friend.

Region 188 displays a list of tools, such as visualization tools, thatAlice can use to view and interact with content and/or preferenceevents.

Story Submission

FIG. 2 is a flow chart illustrating an embodiment of a process forreceiving a story submission. This process may be implemented onpreference server 102. In the example shown, the process begins at 202when information associated with a story is received. For example, insome embodiments at 202, information such as the URL of a story and asummary of the story located at that URL is received.

At 204, one or more checks are performed. As described in more detailbelow, examples of checks include checking to make sure the URL is validand does not, for example, contain a typo; checking for duplicatestories; determining whether the story submission appears on a blacklistor is otherwise to be blocked; determining whether the story is beingsubmitted by a blacklisted user; determining whether the story is beingsubmitted by an anonymous proxy, etc. At 206, it is determined whetherthe story should be accepted.

In some embodiments, if the story submission fails any of the checksperformed at 204, the story is rejected at 210. In some embodiments, athreshold is applied to whether or not a story is accepted at 208. Forexample, a story that appears to be a duplicate may be flagged forreview by an administrator, may be provisionally marked as a potentialduplicate, may be accepted so long as no other checks are failed, etc.In some embodiments, the identity of the submitter is taken intoconsideration when determining whether to accept a story. The decisionof whether to accept the story may be based at least in part on factorssuch as the length of time the user has been a registered user of site116, whether the user has previously submitted inappropriate content,and/or a score assigned to the user.

Typically, the information received at 202 is received through a webinterface, such as a story submission form that can be accessed, byselecting region 158 of interface 150. Other methods of submission mayalso be used, as appropriate. For example, an external website, such asa user's blog could be configured to provide an interface to server 102.Submission may also be automated, such as by syndicating a news feed toa story submission component executing the process shown in FIG. 202. Asdescribed in more detail below, submissions of the information receivedat 202 can also occur through the use of an application programinterface (API), browser plugin, etc.

FIG. 3 is a flow chart illustrating an embodiment of a process forreceiving a story submission. In the example shown, the process isimplemented on preference server 102 and is an example implementation ofthe process shown in FIG. 2. The process begins at 302 when anindication that a story is being submitted is received. Suppose Alicewishes to submit a story. When she selects region 158 of interface 150,server 102 is notified at 302.

At 304, it is determined whether the submitting user is logged into site116. If not, the user is presented with a page at which he or she cancreate an account or log in to an existing account. After completingregistration and/or logging in, the user is directed back to the storysubmission interface (not shown).

A logged in user, such as Alice, is then presented with an interfacethrough which a story may be submitted such as a web form. At 308, a URLis received, such via Alice entering the URL into the form.

System 102 maintains a block list that includes URLs that, e.g., havebeen reported by administrators as spam sites, fraudulent sites, etc. Ifa threshold number of users report a story (such as through region 178of interface 150), the story may be automatically added to the blocklist. At 310 it is determined whether the URL is present on the list ofblocked URLs. In some embodiments, instead of or in addition tomaintaining a list of block URLs, system 102 checks for blocked URLs inconjunction with a third party, such as a commercial anti-spam registry.If the submitted URL appears on the blocked list, the user is presentedwith an error at 312. In various embodiments, the error indicates to theuser the problem with the URL, such as that the URL belongs to a knownspammer. In such cases, the user may be presented with the option ofchallenging the block. In other embodiments, a user submitting a blockedURL is not told why the URL is blocked, or may not be told that the URLis blocked at all. For example, a spurious “system configuration” errormay be presented to the user to help minimize attempts at circumventingchecks.

At 314, it is determined whether the URL can be reached. One way ofperforming this check is to make use of a tool such as curl or wget. Ifthe URL cannot be reached, for example because of a typo (e.g., HTTPStatus Code 404) or because accessing the URL requires a login/password(e.g., HTTP Status Code 401), the user is presented with an error at316. In various embodiments, the user is permitted to revise andresubmit a failed URL without having to restart the process at 302.

Duplicate checking is performed on the URL at 318. In some embodiments,the check performed looks only for exact matches of the URL. In otherembodiments, fuzzy or other matching is applied.

If it is determined that the submitted URL is a duplicate of, or similarto a URL previously submitted to server 302, at 320 the user ispresented with a list of the previously submitted story or stories. Insome cases, the new story submission is a duplicate of an existing storysubmission. In other cases, however, the stories may be distinct,despite sharing a common URL. Such may be the case, for example, with acorporate website that always posts new press releases to the same URL,such as “www.company.com/news.html.” Suppose Alice submits a URL that isalready stored in database 112. At 320 she is asked to compare her storyagainst the other story or stories submitted under that URL (324). Forexample, the interface may present to her a list of the existing storiesmatching her submitted URL in a format similar to story window 164 shownin FIG. 1B. If the story she wishes to submit has already beensubmitted, she can digg the existing story (326), rather than submittinga duplicate. If her story is not a duplicate, she can continue with thesubmission. In various embodiments, considerations such as thesophistication of the user determine whether an exact duplicate URL willbe permitted or whether the user will be forced to digg one of thestories presented on the duplicate list instead of submitting the newstory submission.

At 322, the user is prompted to supply additional information about thestory submission, such as the story's title, a summary of the story, andto which category or categories it belongs. In some embodiments, theinformation collected at 322 is received at the same time that the URLis received (308) and portion 322 of the process shown in FIG. 3 isomitted.

At 328, additional checks are performed on the story. For example, aspam story may escape detection at 310. Such may be the case if the spamwas recently created or is an attempt to unscrupulously drive traffic toa previously legitimate page and is not already present in a blacklist.One check that can be performed at 328 includes applying spam detectiontechniques to the text located at the submitted URL and/or the title orsummary provided by the user. Additional checks may also be employed inaddition to or instead of spam checks at 328. For example, adetermination may be made of whether the submitter (e.g., Alice) isconnecting to site 116 via an anonymous proxy. If it is determined at328 that the submission is spam or should otherwise not be accepted, at330, the submission is rejected. In some embodiments, the submission is“silently” rejected—the user is shown a “successful” dialogue when infact the story is rejected. In other embodiments, the user is presentedwith an error, such as the error presented at 312.

Additional duplication checks are performed on the story at 332. In someembodiments, the submitted title and summary of the story are comparedagainst the titles and summaries of stories already submitted to server102. In some embodiments, the page is crawled and a full text match(such as a MySQL full text search) is performed against the submittedstory and previously submitted stories. In such a case, database 112 isconfigured to store at least portions of the crawls in addition toinformation associated with the story. If it is determined that thestory is a potential duplicate, at 334 the user is presented the optionof digging the story (336) or submitting it anyway.

When a story is accepted at 338, an entry for the story submission iscreated in database 112. Information such as the submission time and theuser who submitted the story are stored and counts associated with thestory, such as its digg score, are set to zero. The story becomesaccessible in the upcoming stories view (e.g., 186 of FIG. 1B).

Information in one or more tables 114 is also updated to includeinformation associated with the new story, for example for use inconjunction with searching, and with visualizations discussed in moredetail below. Additionally, information associated with the submittinguser is modified as appropriate. For example, a count of the number ofstories submitted by the user is incremented, and the story is madeavailable in areas such as the user's profile and the profile areas ofthe user's friends, if applicable.

As described in more detail below, a permalink for the story can beaccessed by visitors to site 116 and contains content assembleddynamically from the information stored in database 112. In system 102,the permalink's URL is created by stripping disallowed characters (suchas punctuation) from the submitted story's title and appending digits asnecessary to avoid collisions. So, for example, if Alice's submittedstory were titled “New Species of Bird Discovered,” once accepted at338, information associated with the submitted story would be accessibleat the URL,“http://www.digg.com/science/New_Species_of_Bird_Discovered.html.”

Also at 338, if the user has specified blogging information, such as ausername and password of an account on a blogging service, the submittedstory is posted to the user's blog. Information such as the summaryand/or title of the story can be automatically copied into the blogsubmission and/or edited by the user prior to submission. For example,if Alice has specified the details of her blog account in her profile(reachable by selecting portion 154 of interface 150), when submittingstory submissions, she can specify whether she'd like the story to alsoappear in her blog. If Alice has not configured her blog settings, theability to blog can be greyed out, hidden, and or explained at 338, asapplicable.

Recording and Reflecting Preference Events

FIG. 4 illustrates an embodiment of a story permalink. The example shownis an implementation of a portion of website 116 as rendered in abrowser. Story 402 was recently submitted to server 102 (26 minutesago), through the process depicted in FIG. 2. When Alice visits thepermalink of story 402, topic region 160 of sidebar 152 automaticallyexpands and highlights the topic with which story 402 is associated (inthis case, “Science”). The story was submitted by David, who also duggthe story. Alice has David listed under her profile as her friend. As aresult, the digg count includes a visual indication 404 that story 402was dugg by a friend. In some cases, Alice and David know each other andhave each other, mutually, on their list of friends. In other cases, therelation may be one sided. For example, David may be a columnist orfamous personality whose opinion Alice values.

The digg score of story 402 is currently two (404) and the story has notmet the threshold(s) required for the story to be promoted out of the“upcoming stories” area.

In the interface shown in FIG. 4, Alice can click digg box 406 toindicate her preference for the story. In some embodiments, additionalactions are taken when Alice diggs a story. For example, if she hasconfigured her blog settings, Alice can specify that stories that shediggs be posted to her blog as she diggs them. Similarly, Alice canconfigure her personal website (e.g., with a JavaScript) toautomatically syndicate recent activities taken in response to stories.

She can report a problem with the story (bury it) by selecting an optionfrom problem dropdown 408. Story reporting options include “duplicate”story (to report that story 402 is a duplicate of another story), “badlink” (to report that the link to the full text of the story isdefective), “spam” (to indicate that the story is fraudulent or spam),“inaccurate” (to indicate that there are factual problems with thestory), and “old news” and “this is lame” to indicate that the story isnot newsworthy. In some embodiments, bury events are anonymous site wideand are not replicated, for example, in a user's publicly accessiblydigging history. One reason for this is to minimize the chances of a“flame war” occurring, for example, when a well known user negativelyrates a story or comment.

As described in more detail below, region 410 displays comments thatusers have made about story 402. Thus far, a total of five comments havebeen left about story 402, two of which were left by Alice's friends.Alice can submit comments by entering information into region 412 ofFIG. 4.

In region 414, Alice is currently viewing a list of all the users whodugg story 402. Suppose David is Alice's friend, but Legolas is not. IfAlice selects friends tab 416, the view in region 414 will change toshow only David's name and avatar icon.

In region 418, Alice is currently viewing a list of the users who haveblogged story 402. Charlie is the only person who has blogged the storyso far and he is not Alice's friend. Therefore, if Alice were to selectfriends tab 420, no names would be shown.

Alice can submit this story to her own blog by entering in optional textin region 422 and selecting region 424. Alice can email the story to oneor more addresses by entering them into region 426 and selecting region428.

As shown, all of the information associated with a particular story(e.g., title/summary of the story, digg score, comments, who has bloggedthe story, etc.) is displayed on a single page. In other embodiments,the information is presented across multiple pages, such as with atabbed view with one or more tabs for each component.

FIG. 5 illustrates an embodiment of an interface to a preference system.The example shown is an implementation of portion 410 of website 116, asrendered in a browser. In the example shown, Alice is viewing commentsassociated with a story. The story currently has eight comments (502),sorted by date. A threshold of −4 diggs or higher has also been applied(518). Thus, comment 516, which has been buried 75 times, is hidden. Inthe example shown, only the header of a buried comment is displayed,along with a link to reveal the hidden comment (522). Additionally, theheader of comment 516 is greyed out to help a user visually distinguishbetween buried and nonburied comments.

Comment 504 was written by Bob, one of Alice's friends, as was comment506 (written by David). In this example, comments written by friends aredistinguished from other comments, such as through having a differentlycolored header. Comments dugg by friends are also distinguished. Thus,while CharlieB is not Alice's friend, his comment (508) is distinguishedbecause it was dugg by Bob, who is Alice's friend, as also indicated bythe inclusion of Bob's name and a star on the header of comment 508. Thenumber of comments left by and/or dugg by her friends are indicated at514.

In the example shown, Bob has written an informative comment, which 18people have dugg. If desired, Alice can digg or bury Bob's comment byselecting the appropriate icon at 520. In the example shown, the diggicon is a green thumb pointing up. The bury icon is a red thumb pointingdown. As described in more detail below, if Alice selects one of theicons, Bob's comment score is immediately updated and the thumbs aregreyed out to indicate to Alice that she's already registered herpreference for Bob's comment.

Suppose Alice finds comment 510 to be off topic or otherwise unhelpful.If she chooses to bury the comment, in the example shown, a variety ofchanges will occur in the interface immediately. The comment score forcomment 510 will decrement by one point. Additionally, comment 510 willcollapse down to just the header, which will grey out. If Alice findsthe poster of the comment, Legolas, a sufficient nuisance, she can blockhim by selecting block icon 524. In this example, if Alice selects theblock icon, she will never be shown any content from Legolas again,site-wide, unless she later chooses to unblock him, such as throughsettings in her profile. Thus, by selecting block icon 524, Alice willnot see comments made by Legolas, stories posted by Legolas, etc.,unless and until she chooses to unblock him.

In some embodiments, if enough people bury a comment, the comment isremoved from the site and/or reported to an administrator. Similarly, ifenough people block a user, in some embodiments, the user is reported toan administrator and/or banned from accessing site features.

If desired, Alice can submit one or more comments of her own. Forexample, she may reply to an existing comment by selecting the replybutton associated with the comment (526), or create a new comment bysubmitting text through region 528. In some embodiments, Alice is givena portion of time during which she may edit the comment, such as withinfive minutes of submitting the comment.

As described in more detail below, when Alice submits or diggs acomment, that preference event is recorded in database 112, her profileand the profiles of her friends are immediately updated, and associatedRSS files are updated.

FIG. 6 illustrates an embodiment of an interface to a preference system.The example shown is an implementation of a portion of website 116reached by selecting region 154, as rendered in a browser. In thisexample, Alice is viewing her profile (hereinafter “interface 602”),which has been subdivided into several tabbed views (604-610). A profileprovides access to a variety of information, some of which may bepublicly viewable, and some of which may be kept private. For example,Alice can change account settings such as specifying her email addressand password by selecting portion 604 of interface 602. Visitors toAlice's profile will be presented with a subset of the informationavailable to Alice. For example, while Alice sees tab 604 being labeled“Profile+Settings,” a visitor to Alice's profile would see tab 604 asleading to Alice's “Profile” only. Similarly, tab 608, which byselecting allows Alice to add and remove friends, is only available toAlice and is hidden from visitors to her profile. Alice can also addfriends by visiting other users' profiles and selecting an “add thisuser as my friend” option located in the profile.

Alice has currently selected to view her friends' history by selectingportion 610 of interface 602. The information presented can be furthercustomized by selecting from subsets of information. For example, ifAlice selects portion 620 of interface 602, she will be presented with alisting of all of the stories that have been dugg by at least one of herfriends. If she selects portion 622, she will be presented with a listof stories that have been dugg by at least one of her friends but havenot yet been promoted. If she selects portion 626, Alice will bepresented with a list of stories submitted by her friends, and byselecting portion 628, Alice will be presented with a list of storiesthat have been commented on by her friends. Other information (notshown) may also be presented in other embodiments, such as a list ofcomments that Alice and/or her friends have dugg.

In the example shown, Alice has selected to view stories “agreed on” byher friends (624). Each of the stories listed in this view have beendugg by at least three of Alice's friends. In various embodiments, Alicecan configure the threshold and specify such information as the numberof friends (or total number of diggs) required for a story to be agreedupon and/or define particular individuals whose digg is necessary for astory to be considered agreed upon, keywords that must be present in thestory, etc. By making use of the “agreed on” view, Alice can readilydiscern the most important stories, even if she has thousands offriends. (I.e., if she sets the threshold to “agreed on by at least 10friends,” and has 1000 friends, the number of stories she is presentedwith is likely to be manageable and especially relevant or interesting.)

Region 616 of interface 602 indicates that four of Alice's friends havedugg story 632. Alice can also see which of her friends have dugg story632 by hovering her input device over the digg score box of story 632.In some embodiments, Alice can interact with region 616, such as bybeing presented with a dialogue that offers to send an email to all ofher friends listed in the region.

By selecting portion 606 of interface 602, both Alice, and visitors toAlice's profile will be presented with Alice's history in a formatsimilar to that currently shown, but limited to activities taken byAlice. Additionally, Alice may “undigg” stories and comments that shepreviously dugg by visiting her history.

All of the views described in conjunction with FIG. 6, such as stories“Agreed On” by Alice's friends can by syndicated as RSS feeds byselecting RSS link 614 on the appropriate page view. In someembodiments, profile visitors (including Alice) are presented with theoption to search (630) all of site 116 for content (634), search Alice'sdiggs for content (636) and/or search diggs made by Alice's friends forcontent (638).

When a user takes certain actions, such as digging a story or burying acomment, the results of that action are reflected immediately, withoutthe user being directed to a “success” page or the appearance of, e.g.,a page refresh occurring to the user. For example, suppose Bob haslisted Alice as his friend. Whenever Alice submits a new story, that newstory immediately appears on Bob's “Friends—Submitted” list and iswritten to the associated RSS file. Similarly, whenever David commentson an article, that fact is immediately reflected under Alice's tab 628as shown in FIG. 6. As described herein, pages served by web module 108include Asynchronous JavaScript and XML (Ajax) components. Othertechniques may also be used to dynamically update site 116 as renderedin a browser as appropriate.

FIG. 7 is a flow chart illustrating an embodiment of a process forrecording a preference for a content contribution. The process begins at702 when an indication that a preference event has occurred is received.For example, when Alice selects digg box 406 shown in FIG. 4, herpreference is received at 702. Other examples of preference eventsinclude submitting a story, burying a story, and commenting on a story.At 704, the preference event is associated with the content contributionand any associated scores are updated as applicable. For example, at704, Alice and story 402 are linked in database 112 and the digg scoreof story 402 is increased in database 112 from two to three. At 706,information associated with the user's profile is updated. For example,as described in more detail in conjunction with FIG. 6, views of Alice'sdigging history (including the friends views of users who have listedAlice as a friend) are updated to include the dugg story and anindication that Alice dugg it. Any RSS files associated with her profileand the profiles of those who have her listed as a friend will also beupdated as appropriate.

Visualizations

FIG. 8A illustrates an embodiment of a visualization interface. Theexample shown is an implementation of a portion of website 116 asrendered in a browser. In this example, interface 800 (also referred toherein as the digg “spy” interface and a “ticker” interface) isconfigured to present a real time visualization of preference eventsoccurring on preference system 102.

In the example shown, a user such as Alice can specify which stories tospy on. For example, she can spy on all stories (802), stories whichhave not yet been promoted (804), or just promoted stories (806).Further specification of a subset of stories can also be applied, asapplicable. For example, in various embodiments, a user can specify akey word that must be present in all stories being spied upon, and/orspy on stories in specified categories (not shown), and/or spy on eventstaken by friends only.

Additionally, a user can specify the types of preference events to bespied upon. In the example shown, Alice has checked (would like to see)all types of activity—new story submissions (indicated by icon 810),diggs (indicated by icon 812), buries (indicated by icon 814), andcomments (indicated by icon 816).

One way of implementing the visualization shown in FIG. 8A is asfollows. As a preference event occurs, it is recorded in database 112.Maintained within database 112 are a main database table and foursmaller tables 114—one for each type of event. The event is alsorecorded (either concurrently with, or on a periodic basis such as byway of an update function) in the respective smaller table thatcorresponds with the event. In some embodiments, filtering is applied sothat, for example, only commenting of registered users is recorded inthe comment table but all commenting is recorded in the main table. Aflag in the main database (e.g., a “do not report this to spy” flag) canalso be set that indicates whether information associated with aparticular story or user should be copied to the smaller tables 114.Alice is a typical user whose diggs are recorded in the main databasetable, as well as the smaller table that records only diggs.

When Alice first visits interface 800 with her browser, and on arecurring basis after that (such as every 20 seconds, or whenever thepool of events is running low), batches of information are retrievedfrom server 102 in a scope commensurate with the options selected (whichdocuments to spy on and for which activities). Specifically, the mostrecent content from each of the smaller tables 114 is retrieved fromserver 102 and stored locally, such as in Alice's browser. AsynchronousJavaScript and XML (Ajax) components in interface 800 cause theinformation to displayed to Alice, for example, at a rate of one eventper second, depending on which information she would like to view. Insome embodiments, Alice can control the speed of the display, such asthrough playback controls 808.

In some cases, such as with heavy digging activity, there may besufficiently more than 20 diggs occurring site-wide during the twentysecond interval between the times that Alice's browser fetches a newbatch of digging information. Thus, after twenty seconds have elapsed,system 102 may have recorded 200 digg events—significantly more than the20 digg events that Alice periodically fetches. In some embodiments,only the most recent 20 actions are fetched. Thus, every twenty seconds,Alice requests the 20 most recent events and will never see anyintervening events.

In other embodiments, the number of events fetched adjusts in accordancewith the speed with which the events are occurring. In such case, all ofthe events are fetched and the rate with which they are displayed issped up (showing one every tenth of a second if there are 200) or sloweddown (showing one every five seconds if there are only four) asappropriate. In some embodiments, a sampling of activity is takenthroughout the period so that if 200 events occur during the 20 secondinterval, a random sample of 20 will be supplied to Alice's browser.

In the example shown in FIG. 8A, Alice has been viewing interface 800for six seconds. Six events (830-840) are displayed, with the mot recent(840) displayed at the top. As a new event is displayed, the alreadydisplayed events are pushed down the display. Thus, for example, at timet1 (when Alice first began watching the interface), only event 830 waspresented. At time t2 (one second later), event 832 was displayed aboveevent 830, pushing event 830 down the screen. At time t3 (one secondafter time t2), event 834 was displayed, pushing events 832 and 830 eachdown one position, respectively.

By consulting the column descriptions (842), Alice can see that event830 was a submission of a new story (818), titled “Scientists DiscoverNew Type of Bird” (822), that the story was submitted by CharlieB(824)), and that the story is currently unpromoted (826) with a diggscore of 1 (820). When event 832 appears in the display at time t2,Alice can see that event 832 was a comment by Legolas on a story titled“How to Make a USB Powered Alarm Clock” that currently has a digg scoreof 33 and has been promoted out of the upcoming stories queue. At timet3, Alice can see that David posted a new story titled “New Species ofBird Discovered.” At time t4, David's story was reported as being aduplicate story (828). The identity of a user burying something is notshown. Instead, only the reason for the bury (such as duplicate story,inaccurate story, old news, etc.) is shown. In other embodiments, otherinformation can be displayed, as applicable.

In FIG. 8A, the displayed digg count for a story is shown as what it wasat the time the event occurred. Thus, when event 838 (a digg of “How toMake a USB Powered Alarm Clock” by Bob) occurs, the digg count of thestory is shown as 33. The next time the story is dugg, the updated scoreis shown, such as when event 840 occurs. If the digg events arehappening sufficiently quickly that some of them are not displayed toAlice, she might see gaps between the scores. For example, if 50 diggsof the alarm clock story occur in the next few seconds, Alice may onlybe presented with the most recent digg and the updated total (e.g., 83diggs).

FIG. 8B illustrates an embodiment of a visualization interface. Theexample shown is an implementation of a portion of website 116 asrendered in a browser being used by Alice. In this example, interface850 (also referred to herein as a “stack” interface/visualization) isconfigured to present a visualization of newly submitted stories. Aftera new story is submitted, such as through the submission interfacedescribed in conjunction with FIG. 3, it is represented in interface 850as a green square falling from the top of the screen, and landing at thebottom. Any stories already existing on the page (e.g., 858, 882, 856,and 854) are shifted to the left to make room for the new story (852).In some embodiments, stories are removed from the left side to makespace for stories on the right side. In other cases, the width of thestories shown decreases to accommodate more stories as they are added tothe interface. The length of time that a user has been viewing interface850 is shown here as a timeline, with the time that Alice first startedviewing interface 850 on the left (866) and the current time on theright (868).

In the example shown, eleven new stories have been submitted since Alicebegan viewing interface 150. Statistical information such as the numberof stories submitted, the rate with which they are being submitted,etc., is indicated at 870. As preference events associated with thestories displayed in interface 850 occur, they are also indicated ininterface 850. For example, when a story is dugg, the event isrepresented by a digg icon, such as the one shown at 812 in FIG. 8Afalling from the top of the screen and onto the heap of thecorresponding story, increasing the size of the heap if it is a digg,and decreasing the size of the heap if it is a bury. For example, at860, a digg of story 858 is shown falling down onto the story'sgraphical representation and will increase the height of the story boxwhen it lands. A variety of indicators, such as colors and avatars canbe used to indicate the occurrence of preference events in addition toor instead of the icons shown in FIG. 8A.

At 864, a bury of story 882 is shown falling down onto that story'sgraphical representation and will decrease the height of the story boxwhen it lands. In the example shown, the bury is indicated by the buryicon shown at 814 in FIG. 8A. The identity of the user burying the storyis not shown (as it can be in the case of other preference events), butby hovering her mouse over bury 864, Alice is shown a dialogue thatincludes the reason that the bury was submitted (e.g., “spam”).

In some embodiments, additional elements are included, such as theanimation shown at 880 (of a missile about to strike heap 882), andindications of who is taking the action. For example, diggs 860 and 862are being performed by friends of Alice. She is alerted to this fact bybubbles accompanying the digg action being performed that indicate theirnames and/or avatars. The look of interface 850 can be skinned in someembodiments—Alice can specify that she desires the interface to have amilitaristic theme, such as in the example shown, or other themes, suchas ones in which animals “eat” stories or multiply.

The relative popularity of newly submitted stories is indicated by therelative heights of the stories shown in interface 850. For example,story 884 is a very popular story, while story 856 is not.

In the example shown, only newly submitted stories are shown. Interface850 can also be scoped to represent other portions of activity. Forexample, Alice can specify that she wants to observe only actions takenby her friends (or groups of her friends), but across all stories, newor old. Alice can also specify that she wants to observe only actionsthat include a particular keyword, actions occurring only in particularcategories or subcategories, etc. Alice can also specify particularstories she wishes to monitor, such as by selecting a link on the page'spermalink that reads, “add this story to my incoming view.” Alice canalso “pin” the stories shown in interface 850. When she hovers her mouseover a particular story shown in interface 850, one element that isrevealed is a pushpin icon which, if selected, causes the story toremain in region 886 of interface 850, and/or be added to a list offavorites (878).

A variety of graphical tools are shown on the left hand side ofinterface 850. They include charts of information such as which storiesin the last hour have been most popular 876, the relative rankings ofstories that Alice is monitoring (has pinned) 878, a more comprehensiveview (i.e., including information predating Alice's current interactionswith interface 850), etc. At 872, the story entry 170 of a story thatAlice hovers her mouse over, such as story 854, is displayed and she caninteract with the story entry 170 such as digging it or burying itaccordingly.

FIG. 8C illustrates an embodiment of a visualization interface. In theexample shown, a user may switch between different visualization styles(e.g., implementations of “stack” and “swarm”) by selecting from amongchoices provided in region 888. The visualization shown in FIG. 8C hasmultiple modes which can be selected from in region 889. When “allactivity” is selected, all preference events occurring in all areas areincluded in the visualization. When “popular stories” is selected,stories that have been promoted to the front page are shown, arranged inthe order in which they were promoted. When “upcoming stories” isselected, the most recently submitted stories are shown. In someembodiments, information such as a color scale is used to help depictwhen newly submitted stories have associated preference events. Forexample, if a newly submitted story has twenty or more diggs, therepresentation of the story may be colored bright red, indicating thatthe story is rapidly gaining interest.

The interface shown in FIG. 8C also includes a region 893 in which auser may select between seeing a visualization depicting the activitiesassociated with individual stories (by selecting “Stories”) or by seeinga visualization depicting the aggregated activities associated with thecategories to which stories are assigned (by selecting “Topics”). In theexample shown in FIG. 8C, the “Topics” view is selected, and thepreference events being visualized are shown relative to the topicsrather than individual stories.

The interface shown in FIG. 8C also includes pause (890) and zoom (891)controls. The current amount of zoom is indicated in region 892. Byusing zoom control 891, groups of stories (e.g., the most recentstories) can be focused on, or the visualization can be pulled back fora broader view. Pause control 890 can be used, for example, to assist inmore readily focusing on a specific story when a great deal of activityis occurring and the visualization would otherwise change rapidly. Whenin the “popular stories” mode (889), the zoom control can be used asfollows. If the visualization is zoomed all the way out, the user isable to see the n most recently promoted stories, and get a sense ofwhich stories continue to have a significant amount of associatedpreference events, even if they are no longer on the front page. Thevisualization provides information on which stories are active, even ifthey were not recently submitted. The value of n can be configured bythe user and/or by a site administrator. An example default value of nis 100.

FIG. 8D illustrates an embodiment of a visualization interface. In theexample shown, Alice has selected a story from a stack visualizationinterface, such as by clicking on one of the stacks shown in FIG. 8B(e.g., 884 or 856). In doing so, Alice is presented with additionalinformation about that story, such as who dugg it, how many comments ithas, etc. Links to the permalink page, etc., are also provided. In theexample shown in FIG. 8D, a sparkline-style graph is also provided thatshows a more detailed hour-by-hour display of activity on that story,including the frequency and magnitude of preference events. In variousembodiments the information shown in interface 894 is configurable. Forexample, while a default view may show the last 48 hours of activity,the user (or an administrator) may be able to specify additional ranges,and/or the default range may be selected based on how much activityassociated with the story has occurred. For example, a story with a lotof recent preference events may default to a 12 hour range, while an oldstory may show a histogram of all activity over all time.

FIG. 9 illustrates an embodiment of a visualization interface. Theexample shown is an implementation of a portion of website 116 asrendered in a browser. In this example, interface 900 is configured topresent a visualization of the genealogy of the diggs (also referred toherein as a “tree view”) of a story on preference system 102.

In the example shown, story 902 was originally submitted by the userDavid. When he successfully submitted the story, it appeared in hisprofile, as well as in the Friends History (e.g., at 610 of FIG. 6) ofthe several people who have listed David as their friend. Suppose tenpeople have David as a friend, including users 904, 908, and 912 andseven users not pictured. After David submitted the story, friends 904,908, and 912 dugg story 902, either through their own friends pages, orthrough David's profile. They are displayed in interface 900 asconnected to David. Users who have David listed as a friend who did notdigg the story are not displayed in interface 900. Users 906 and 910 donot have David as a friend, but dugg the story through visiting hisprofile. As a result, they are also shown connected to David.

When users 904-912 dugg story 902, that action was recorded in theirrespective user profiles as well. Visitors to their profiles, and thosewho list them as friends who digg the story will be shown connected tothem, the way they are shown connected to David. If expansion tab 914 isselected, interface 900 will continue to provide detail down the tree(those who dugg story 902 through user 916, and so on).

One use of the tree view is that users can trace how their friendslearned about stories and meet new friends. For example, if Alicenotices that Bob diggs a lot of cryptography stories, she can determinewhere Bob diggs them from—does he submit the stories himself, or is hemainly digging stories submitted by CharlieB—and add new friends (suchas CharlieB) as appropriate.

FIG. 10 illustrates an embodiment of a visualization interface. Theexample shown is an implementation of a portion of website 116 reachedby selecting region 186 of FIG. 1B as rendered in a browser. In thisexample, Alice is viewing upcoming stories, which may be displayed in avariety of ways. If she selects region 902, Alice will be presented withupcoming stories in a format similar to that shown in the story window164 shown in FIG. 1B (including one or more story entries 170). In theexample shown, Alice has selected to view the upcoming stories in acloud view by selecting tab 904. In this view, the title of each storyin the upcoming queue is visualized as a function of the number of diggsit has. Stories with few diggs are shown in a very small font, and maybe colored in a subtle manner, such as by being displayed in grey.Stories with many diggs are shown in a very large font and may bedisplayed in another color, such as red or black. Stories dugg byfriends are also shown in a different color, such as green, irrespectiveof number of diggs. In some embodiments, additional information isreceived from the interface shown in FIG. 10 by taking an action such ashovering a mouse over a story title. In such case, information such asthe current digg score of the story, which if any friends have dugg thestory, and/or the story entry 170 of FIG. 1B is shown.

Which stories will appear in the cloud view can be configured, such asby selecting one or more categories to view or limiting the view tostories dugg by friends. The cloud view can also be sorted by a varietyof factors. As shown, the newest (most recently submitted) stories areshown at the top, and older stories are shown at the bottom of FIG. 10.If the stories were sorted by most diggs, then stories rendered in thelargest font would appear first and stories rendered in the smallestfont would appear last. Other sorting methods may also be used, such asby sorting by most or least comments.

FIG. 11A illustrates an embodiment of a visualization interface. Theexample shown is an implementation of a portion of website 116 reachedby selecting the appropriate portion of region 188 of FIG. 1B asrendered in a browser, and is an example of a swarm interface (alsoreferred to herein as a “swarm visualization”). In this example, Aliceis viewing upcoming stories. Users are shown represented by their avataricons, or by more generalized shapes. As they digg a story, their iconis shown “swarming” around the story in real time—the avatar moves nearthe story the user is digging, as do the avatars of the other userscurrently digging the story. In some embodiments, the size of the user'savatar (or other representation of the user) increases and decreasesbased on the number of stories they are currently digging.

In some embodiments, only recent activity is shown—such as diggs in thelast 10 minutes. Stories with more activities (such as diggs andcomments) will appear larger than stories with fewer activities. In someembodiments, additional information is received from the interface shownin FIG. 11A by taking an action, such as hovering a mouse over a storytitle. In such case, information such as the current digg score of thestory, which, if any friends have dugg the story, and/or the story entry170 of FIG. 1B is shown. The links between stories can also be shown,indicating, for example, that several of the same people that dugg aparticular first story also dugg a second story, by connecting the twostories with a line. Indicators such as the color or width of the linecan show how strong or weak the connection is between the stories.

FIG. 11B illustrates an embodiment of a visualization interface. Theexample shown is an implementation of a swarm visualization. In theexample shown, stories are represented as circles, with the title of thestory in the center of the circle. Users “swarm” around the stories whenthey indicate a preference for the story, such as by digging it (and/orcommenting on it, as applicable). Every time a story is dugg, thestory's circle increases in size. Thus, the bigger the circle, the moreactive the story is. In the example shown, story 1130 is very popular,while story 1132 is less popular. Story 1134 has very few associatedpreference events.

As users digg more stories, they move from circle to circle, and alsoincrease in size. For example, a very large user might represent aperson who is not taking much time to read stories, but is insteadmerely rapidly indicating preferences. In the example shown in FIG. 11B,the user “Bob” (1136) has recently indicated preferences for manystories, while other users (e.g., user 1138) are less active. In theexample shown, stories are initially randomly placed within theinterface. As preference events associated with the stories occur, theirpositions change depending on who is digging (commenting, etc.) on them.For example, stories that are closer together indicate that they arebeing dugg by the same users, and by hovering a mouse over the story,such connections between stories are revealed.

Different modes of the swarm visualization may be presented by selectingone of the options in region 1140. For example, if the “all activity” isselected, circles representing stories and diggers are quickly removedfrom display if no associated preference events are occurring/beingmade, respectively. When the “popular stories” mode is selected, thedisplay is initially loaded with the n stories most recently promoted tothe front page. As new stories are promoted, they appear in thevisualization, and the (n+1)th story is removed. The value of n may beconfigured, e.g., by a user or an administrator. In some embodiments nis 35. When the “upcoming stories” mode is selected, the n most recentlysubmitted stories each receive a circle. In some embodiments n is 30.

FIG. 11C illustrates an embodiment of a visualization interface. In theexample shown, Alice has selected a story from a swarm visualizationinterface, such as by clicking on one of the story circles shown in FIG.11B. In doing so, Alice is presented with additional information aboutthat story, such as who dugg it, how many comments it has, etc. Links tothe permalink page, etc., are also provided.

In the example shown in FIG. 11C, the lines between stories indicatecommon diggers between those stories. The more diggers in common that astory has, the thicker the line. For example, story 1150 and 1152 haveconsiderably more common diggers (as indicated by line 1156) than story1150 and 1154 do (as indicated by line 1158).

Alice may also override the default amount of time a particular storywill be displayed in (e.g., in the interface shown in FIG. 11B) byselecting either region 1150 or 1162 of the interface shown in FIG. 11C.Thus, for example, to prevent story 1150 from being removed when a newstory is displayed, Alice may select region 1160. To immediately removethe story from the interface irrespective of when it might otherwisehave been removed, Alice may select region 1162.

Additional Embodiments

In some embodiments, a plugin and/or add-on to a computer programproduct is used to provide digging (and/or burying, commenting, andsubmitting) functionality. The plugin/add-on is associated with aninterface to server 102 which may include functions that determinewhether a permalink already exists for the submission, and invokeprocessing of a new submission, a comment, etc., as appropriate.

For example, a plugin to a web browser (e.g., a Firefox extension) canbe configured to offer a user the ability to digg an item directly fromthe context in which it is encountered without having to visit thesubmission interface described in conjunction with FIG. 3 or a permalinksuch as the one shown in FIG. 4. For example, a notification embedded ina page or overlayed such as by the browser can indicate whether the pagea user is currently browsing has been submitted as a story contributionyet. If not, a user can interact with the notification, such as byclicking on an interface that reads, “this page has not yet beensubmitted to digg.com, submit it now.” Similarly, if the page hasalready been submitted, such as by a different user, the notificationmay take a variety of forms, such as an overlay of the current diggscore (172) and a digg box, or a change, for example, in the backgroundcolor, or some other element of the page.

Configurable dropdowns and/or overlays can also be provided to alert auser of certain activity. For example, the user can set an alert toreceive notification when new stories having certain keywords aresubmitted to server 102. Notification can also be provided for friends'digging activities as they occur, such as that a friend has just dugg astory or commented on a product.

As used herein, content contributions are pointers to content (e.g.,news articles and podcasts) that is stored outside of preference system102, typically by a third party. In some embodiments, users submit thecontent itself (e.g. the full text of articles, and the audio file)rather than or in addition to a pointer to the content, and thetechniques described herein are adapted accordingly. The terms“content,” “content contribution,” and “pointers to content” are usedherein interchangeably. As described in more detail below, contentcontributions are not limited to news articles. Other content (such asproducts, services, songs, sounds, photographs, and video) can besubmitted, dugg, buried, and commented on and the techniques describedherein can be adapted as appropriate. Preference events taken on thosetypes of content may likewise be associated with a profile and sharedwith friends in a manner similar to that described, for example, inconjunction with FIG. 6.

FIG. 12A is an example of a content contribution. The example shownrepresents a restaurant submission. The name of the restaurant (1200) isincluded, as is information such as who submitted the restaurant, theURL of the restaurant, the type of cuisine it serves (1202), and thegeneral location of the restaurant (1204). Users may perform suchactions as searching for restaurants by cuisine type and/or location,and limiting results to ones having a threshold number of diggs.Restaurants having no or few diggs can be displayed as “upcomingrestaurants,” separated from “promoted restaurants” which have diggscores exceeding a threshold. Users can also supply additionalinformation about their preferences for the reference, such as bysupplying one or more tags (1202) that indicate attributes such as“ambiance” or signature dishes. As described in more detail below, whichfields/tags are collected at submission time (and which, if any, can beadded subsequently) and shown can be configured as appropriate dependingon the type of content. For example, in the case of a product, a stockphoto of the product may be included.

FIG. 12B illustrates an embodiment of an interface to a preferencesystem. In the example shown, the interface unifies a user's preferencefor things across multiple genres of content. For example, the user candigg for news (1250), videos (1252), and restaurants (1254) all throughthe same interface. As described in more detail below, the friendsfeatures described above can also be used in conjunction with othertypes of content contributions. For example, using the interface shownin FIG. 12B, a visitor to Alice's profile can learn which news storiesshe's been digging as well as learn which restaurants she diggs ordoesn't digg. Similarly, Alice can customize the views of each of thetabs (1250, 1252, 1254) to display only restaurants her friends ofagreed on, restaurants nearby (e.g., by selecting a region on a map orentering a ZIP code) that at least one friend has dugg, etc.

FIG. 12C illustrates an embodiment of an interface to a preferencesystem. In the example shown, digging functionality has been combinedwith mapping functionality. When a user searches a map, such as aweb-based map service, for nearby restaurants, entries on the mapinclude an indication of the number of diggs a business has had and theability to digg or comment on the business directly from the mapinterface.

FIG. 13A illustrates an embodiment of an interface to a preferencesystem. The example shown is an implementation of a portion of website116 which includes the ability to submit, digg, and comment on products(including software), as rendered in a browser. In this example, Alicehas selected to view products agreed on by her friends (1322).

Alice can submit a new product review by selecting portion 1302 ofinterface 1300. She can view products in one or more categories byselecting the appropriate portion of region 1304. Portion 1306 ofinterface 1300 displays the recent activities of Alice's friends in adashboard format.

Region 1326 of interface 1300 indicates that four of Alice's friendshave dugg product 1324, the ACME MP3 player. Alice can also see which ofher friends have dugg product 1324 by hovering her input device over thedigg score box of product 1324. In some embodiments, Alice can interactwith region 1326, such as by being presented with a dialogue that offersto send an email to all of her friends listed in the region. In someembodiments, additional actions can be taken with product 1324. Forexample, Alice may be presented a “buy this product now” icon or link.

All of the views shown in FIG. 13A can be syndicated as RSS feeds byselecting RSS link 1320 on the appropriate page view. For example, ifAlice is a professional critic, users and those who choose not to useweb site 116 on a regular basis can syndicate comments that she makes onproducts, etc.

In some embodiments, profile visitors (including Alice) are presentedwith the option to search (1308) all of site 116 for product keywords(1310), search Alice's diggs for product keywords (1312), and/or searchdiggs made by Alice's friends for product keywords (1314). For example,a visitor to Alice's profile can search for MP3 players that she hasdugg or commented on. In some embodiments, search interface 1308includes the ability to filter results on meta information such asregions for DVDs, languages for books, etc. In some embodiments, views(and searches) can be limited by other factors, such as location(distance from Alice), availability (whether a product is in stock andhow quickly it can arrive), etc.

FIG. 13B illustrates an embodiment of an interface to a preferencesystem. The example shown is, as rendered in a browser, animplementation of a portion of website 116 that includes the ability tosubmit, digg, and comment on products. In this example, Alice hasselected to view products in the category, MP3 player, from a larger setof categories, such as those listed in region 1304 of FIG. 13A.

In the example shown, each product listing (1302, 1304) includes aphotograph of the MP3 player (1306), as well as a digg score/digg box(1308), title, description, etc. (1310). The MP3 players shown in thisexample are sorted by popularity.

On the right hand side are assorted graphs (1312, 1314) of informationassociated with the products shown. Graph 1312 compares the popularity(e.g. digg scores, number of comments recently made, etc.) of differentMP3 players against each other over time so that trends such as whichones are gaining in popularity and which ones are decreasing inpopularity can be visually determined.

In the example shown, the Acme F242 player is more popular than the Beta10 player. In some embodiments, the frequency with which a user visitspreference system is considered when determining the popularity of aproduct. For example, suppose the Beta 10 player has 165 diggs, 25 ofwhich were made by users who have not visited the preference system in 3months. In some embodiments, the diggs of those 25 users are expired.The product will remain listed in the absent user's profiles, but theirdiggs will not be included when calculating the popularity of theproduct.

Users also have the ability to undigg a product to indicate that they'vemoved onto something new. For example, suppose Alice currently has aBeta 10 player and is interested in upgrading. If she purchases an AcmeF242, she can visit her profile to undigg the Beta 10 and digg the AcmeF242. Her actions—undigging the Beta 10 and digging the Acme F242instead—will also be reflected in graph shown at 1312. For example, onthe day that she undiggs the Beta 10, its position along the verticalaxis will be decreased. On the day that she diggs the Acme F242, theAcme F242's position on the graph will similarly increase.

Graph 1312 also includes indications of the individual users who aretaking digging and undigging actions. For example, when Alice hovers hermouse over region 1318, she can see that a user, Mark, dugg the AcmeF242. Indications of actions taken by her friends are also included ongraph 1312. For example, regions associated with friends' diggs of theAcme F242 are highlight in green, or with avatars, or other indicatorsthat her friends have indicated preferences at a particular time. Forexample, Alice can use graph 1312 to determine that David dugg the Acme242 two months after Charlie dugg the Beta 10.

The information shown in FIG. 13B can also be generated based on one ormore searches in addition to or instead of tabbed browsing. For example,Alice could perform a search of “popular MP3 players at least one of myfriends owns” and see the information shown in FIG. 13B as a result.

In some embodiments, demographic information is displayed. For example,in graph 1314, the popularity of a particular MP3 player is broken downby assorted groups such as teens and adults, or girls and boys.Demographic information can similarly be included in a search so that,for example, a parent shopping for a present for his or her child canlocate the “hottest MP3 players among teenagers this week,” and the“most popular movie for women aged 20-30,” through a search interfaceconfigured to accept such queries.

FIG. 14 illustrates an embodiment of an interface to a preferencesystem. The example shown is an implementation of a portion of website116 which includes the ability to submit, digg, and comment onphotographs and video, as rendered in a browser. In the example shown,photograph 1402 was dugg by a friend, as indicated by banner 1404. Byselecting digg box 1406, a visitor can indicate a preference for thephotograph shown. In some embodiments, visitors indicate theirpreference for content such as video 1408 by selecting an icon such asicon 1410.

The content shown in interface 1400 can be presented in a variety ofways. For example, video content may be represented as an icon, such asthe filmstrip icon shown at 1408. A screen shot of the first frame ofthe video may also be shown, and interactions, such as hovering a mouseover region 1408 could trigger actions such as causing the video to beplayed in the browser.

In some cases, it may not be possible to embed the content directly intothe interface shown in FIG. 14. In such a case, the video is shown in aformat similar to story entry 170 (1416), and a preview button 1414 isincluded. When preview button 1414 is selected, a video player 1412automatically slides out in which the video can be displayed.

Permalink pages such as the one shown in FIG. 4 can be adapted forphotograph and video content as appropriate, and users may comment,blog, and take other actions with respect to visual and other content(such as songs) as appropriate.

Cohort Detection

A nefarious individual may create multiple personas all under hiscontrol or offer money to other users in exchange for their complicityin digging stories. A group of users (e.g., in the same dorm, company,or other organization) may attempt to digg stories favorable to thatentity, etc. Groups of users banding together in such a fashion arereferred to herein as “cohorts” and “associates” interchangeably.

In some cases, cohorts may be “friends” on the site, with profileslinked to one another and may visit each other's profiles and click onall of their friends' stories. Cohorts may also organize in a moreclandestine fashion, relying on tools such as instant messages and email(e.g., out of band) to agree to digg one another's stories so that theyare promoted (e.g., reach the front page) sooner. In some cases, atleast some cohorts may be acting subconsciously or otherwise beinnocent. For example, a user, “FanBoy” may always digg the posts of auser, “RockStar,” without ever reading them, because FanBoy likesRockStar and wants to see RockStar's submissions always be promoted,irrespective of their merit.

In general, typical users making typical use of “friend” features maylegitimately visit their friends' profiles and digg some of the storiestheir friends have dugg. Legitimate friends typically do not digg oneanother's stories in an orchestrated, continual fashion. As described inmore detail below, a variety of techniques can be employed to detect theactivities of the aforementioned “digging associates,” and to lessen theimpact of concerted digging and group/friend-think on the promotion ofcontent, while not penalizing legitimate friend-related activity.

FIG. 15 is a flow chart illustrating an embodiment of a process fordetecting associates. In some embodiments the process shown in FIG. 15is performed by cohort detection module 120 and promotion engine 118working in conjunction. The process begins at 1502 when a plurality ofindications of preferences for a content item is received from aplurality of users. Typically, the processing performed at 1502 occursover a period of time. For example, suppose a user, Bob, submits a storyto preference system 102. Alice diggs it a few minutes later, a user,Charlie, diggs it a few minutes after that, and so on. Bob's submissionand Alice and Charlie's diggs, collectively, are received at 1502.

At 1504, the preference events received at 1502 are accumulated. Forexample, in some embodiments, information stored in database 112 isaccessed and all of the preference events associated with the storysubmitted by Bob are retrieved. In various embodiments, the processingperformed at 1504 executes as part of a batch process. At 1506,associated events are detected. Several examples of ways in whichassociated events (e.g., events submitted by cohorts) can be detectedare described in more detail below. At 1508, the effect of theassociated events is reduced, prior to assigning a status to the item.For example, prior to determining that content should be promoted, theeffects of associate digging are removed. As another example, theeffects of cohort burying are removed prior to determining that contentshould be removed (e.g., so that legitimate content is not removed).

In some embodiments, cohort detection is performed as follows. Once aweek, the preference events for the preceding week are retrieved fromdatabase 112 and evaluated. In some embodiments the entire store ofpreference events for the time period (e.g., week) is evaluated. Forefficiency, evaluation may also be limited to include just the behaviorof friends, to include only preference events associated with unpromotedcontent, etc.

A list of cohorts is determined and stored in database 112. Whendeterminations about the promotion (and/or removal) of candidate contentare made for the following week, the list of cohorts is consulted. Forexample, if cohort digging has been detected for a particular item, theimpact of the cohort digging is reduced. One example of such a reductionis by discarding from consideration the digg counts provided by thedetected cohorts. Another example of such a reduction is to increase thenumber of digg counts required for the content to be promoted. At theend of the week, the cohort list is updated based on the preferenceevents recorded during the week.

EXAMPLE Groups

Group cohorts are associates that have very common habits. One exampleis users that bond over a particular micro-community or area ofinterest. People who are intensely in favor of a particular politicalcandidate, or those passionate about a niche technology market, or liveon the same dorm floor are examples of users that might form groups.Typically, those users are seemingly random users (with different IPaddresses—that log in at different times of the day, etc.) who have avery high correlation factor of the stories they digg on a regularbasis.

Two or three or four users could in fact randomly happen to digg thesame two or three stories, however for a larger group to digg a largernumber of stories in common, a statistically probable correlationbetween their digging activities may be sufficient to mark those usersas group diggers. Multiple techniques can be used to detect groups ofdiggers.

Suppose that cohort detection includes evaluating, for each user, thebehavior of that user's friends. In this example, suppose Alice has tenfriends and has dugg (or submitted) a total of twenty stories in thelast week. In such a scenario, at 1504, the preference events of Alice'sfriends with respect to those twenty stories are evaluated. For each ofthe twenty stories dugg/submitted by Alice, a count is made of thenumber of Alice's friends that also dugg that story. The first story wasdugg by three of her friends (Charlie, Dan, and Eve). The second storywas dugg by five of her friends (Charlie, Dan, Eve, Joe, and Ted). Thethird story was dugg by two friends (Dan and Eve), and so on, with thetwentieth story being dugg by eight of Alice's friends. In someembodiments the “Agreed Upon” feature is used in conjunction with theprocessing shown in FIG. 15.

Stories with a high number of common friends might indicate cohortbehavior with respect to those stories. Variables such as how manyfriends need to agree across how many stories are configurable. Thevariables may also be dynamic and depend on context such as the categoryof the story, or—in the case of other content—the type of content. Asone example, a rule may be specified that if a group of the same 50% ofa user's friends agree on 20% of a user's submissions/diggs, that groupis a cohort group and its diggs should not be counted in contentpromotion evaluations. In Alice's case, if 5 of her friends agree on 4or more stories, those 5 (or more) friends' diggs, as well as Alice's,will not be counted. An evaluation of Alice's friends' activity for theweek indicates that Dan and Eve routinely digg the same content thatAlice does—and that if Alice has dugg a story, and Dan and Eve have alsodugg that story. However, since Dan and Eve represent only two ofAlice's ten friends, no cohort group will be designated for Alice, Dan,and Eve.

In some cases a user may belong to multiple groups. For example, supposeAlice has an agreement with her friends that she will digg all of theirstories if they digg hers. She also has an arrangement with a politicalparty that she belongs to that she will always promote stories that arefavorable to a particular candidate (irrespective of who submitted thestory). Suppose there are 200 members of the political party on thesite. Only some subset of those members may be detected as being cohorts(i.e., the ones who digg frequently). Nonetheless, by detecting thecollective behavior of that subset, the stories about that candidate areless likely to be promoted due to Alice and her cohort's activities thanthe activities of average users.

Another example of a technique for detecting cohorts follows. A table ofactivity for a time period is constructed in which users form the x axisand stories (or other content) form the y axis. For example, usersactive over the last 7 days and the stories promoted over the same timeperiod are used as the respective axes. Each cell in the table ispopulated with information indicating, e.g., the friends of the userthat also dugg that story. In some embodiments the cell is populatedwith all of the other individuals that dugg that story irrespective offriend status. Clustering techniques can then be applied to determine,for a given row, whether there are other users that consistently appear.Suppose out of the 20 stories Alice dugg, the digging history of 10 ofthose stories shows that in addition to being dugg by Alice, users A2,A3, A4, A5, A6, and A7 also dugg each of those 10 stories. A cohortgroup consisting of Alice and users A2-A7 can be formed and stored. Aspart of the promotion evaluation process (described in more detailbelow), each of the diggers of that candidate story is checked to see ifthat user is present in the cohort table. If so, it is determinedwhether any of that cohort's cohorts have also dugg the story. If so, athreshold is applied, and it is determined whether cohort behavior isindicated or not. For example, suppose Alice diggs story S5. The usersA2, A3, A5, A6, and A7 have also dugg story S5, but user A4 has not.This likely indicates cohort behavior because all but one of the membersof a cohort group have been detected as having dugg the same story.Suppose now that Alice diggs story S6 and only A2 and A7 digg it. Inthis case, cohort behavior (on the part of Alice and the rest of theA2-A7 group) is not likely present. The process is repeated for othercohort groups. And, for example, it may be determined that another group(e.g., including a different group of users) is implicated by using thetechniques above.

Suppose Alice is determined to be a member of three cohort groups. Thefirst group consists of Alice and users A2-A7 as above. The second groupconsists of Alice and a second set of users, and the third groupconsists of Alice and a third group of users. Now suppose a story S7 isotherwise able to be promoted and is being evaluated for cohorts. Eachof the diggers of the story is searched for in the cohort list. If noneof the diggers is present in the cohort list, the story may be promoted.Suppose Alice dugg story S7. It is determined that Alice is in thecohort database. Suppose user A2 dugg the story, but none other of thefirst group (Alice and A2-A7) dugg the story. A collective action on thepart of Alice's first group is not implicated because an insufficientnumber of group members (e.g., compared to a threshold) dugg that story.In some cases the threshold required is all of the members of Alice'sgroup. In other cases, a looser match may be performed against Alice'sgroup members, e.g., to accommodate situations such as where one or twoof the group members is out of town (and will not be digging as acohort) when all the other members do. The rest of Alice's groups arealso checked—and, if a threshold number of members in either of herother groups also dugg story S7, their collective action can bemitigated accordingly.

In various embodiments, the cohort groups listed in the cohort table isperiodically updated to prune and add new groups. In some embodimentsthe updates are performed across all data, across data that is new sincethe last time cohort groups were detected, or using a rolling sample ofdata.

In some embodiments a pool of candidate cohorts is maintained andadditional candidates are include or rejected from the pool. Forexample, suppose Alice dugg 25 stories during a sample time period. Foreach of the other users that were active during that time period, atally is made of how many stories each of those users had in common with(also dugg with) Alice. The tally is sorted and users with talliesexceeding a certain threshold are designated as cohorts of Alice(assuming she has any).

The technique can also be used to recommend other stories, by findingother users who have an overlap of stories dugg (and therefore a “high”correlation factor).

In various embodiments, genetic algorithms are used to determine groupsof cohorts.

Example Weighting a First User's Agreement with a Second User

The following is an example of a technique for addressing the blindfollowing of one user by another. In the technique, each subsequent diggby a user, to another user's submitted stories, within a time period, ismade to be valued less (e.g., less than one vote, and/or as a reductionto the “user reputation score” discussed in greater detail below). Forexample, within a time period (e.g. 30 days), each subsequent time thatAlice diggs one of Bob's stories, Alice's digg is attributedprogressively less value. The first time Alice diggs a story submittedby Bob, within a rolling 30 days, it's worth one point (or, other fullvalue ascribed to Alice). The second time Alice diggs a story submittedby Bob, that second story receives a lower score. The third time Alicediggs a story submitted by Bob, that third story receives an even lowerscore. If Alice diggs many of Bob's submissions during the 30-dayperiod, it is possible that some of those stories may receive zero valuefrom Alice. Any other stories dugg by Alice, and not submitted by Bob,will receive Alice's full value.

Example IP Address

In various embodiments, preferences indicated for the same content byusers that share an IP address are counted less and less, e.g. using avariant of the technique described above. In the modified technique, thefirst digg from the IP address for a particular content is given a fullcount. Each subsequent digg from the same IP contributes exponentiallyless toward promotion. In this scenario, users creating multipleaccounts and using them from the same machine, and users from the samecompany or school digging the same content will require greaterdiversity. However, the activities of two users legitimately sharing thesame computer will typically still count, in full, toward the site.

Example Statistical Analysis

Yet another way of performing cohort detection is to examine everycontent item, and every preference event for that item, and to comparethe results to all other content items to determine how much overlap/howmany diggers they have in common.

Suppose a first story (that has 10 diggs) is compared to a second story(that also has 10 diggs). The two stories would have a 100% correlationfactor if they were dugg by the exact same set of 10 users. If the twostories had nine diggers in common, the story still likely indicatesgroup behavior, whereas if the two stories had four diggers in common,autonomous decisions on the part of the diggers are likely to have beenmade.

Correlation factors can be calculated a number of ways includingstandard similarity factors used in mathematics such as Jaccardsimilarity coefficients and the Jaccard distance between two sets ofitems. For example, A and B can be represented by two different user'ssets of diggs over a given timeframe. The following algorithm computesthe Jaccard distance—or similarity factor between two user's diggingpatterns:

${J_{\delta}( {A,B} )} = {{1 - {J( {A,B} )}} = {\begin{matrix}{{{A\bigcup B}} - {{A\bigcap B}}} \\{{A\bigcup B}}\end{matrix}.}}$

Once all correlation factors are calculated for how a particular storyoverlaps with every other story, a normal standard deviation breakdownof the correlation factors can be determined. The majority of storiesare likely to fit inside the middle of the bell curve, while one or twostandard deviations out would represent some highly correlated stories.In some embodiments, stories more than one standard deviation outindicate group digging behavior for stories over a minimum threshold(e.g., 20 diggs).

Promoting Content

As mentioned in conjunction with FIG. 1B, a variety of techniques can beused to determine when content, such as a story, should be “promoted”out of the upcoming area or similar area and shown on the main page orother appropriate location.

FIG. 16 is a flow chart illustrating an embodiment of a process forpromoting content. In some embodiments the process shown in FIG. 16 isperformed by promotion engine 118. The process begins at 1602 when oneor more candidate content items is determined. For example, at 1602, allof the items in the upcoming stories pool might be selected. Optionalpruning of the candidates may also be performed at 1602. For example,stories more than 48 hours old, reported as violating a term of service,or otherwise meeting pruning criteria may be removed from the set ofcandidate content items at 1602.

At 1604, user reputation scores are received, such as from database 112.A user reputation score approximates how good a particular user is atsubmitting and/or locating “good” content, and in some embodiments howgood a particular user is at identifying “bad” content. One way ofdetermining a user reputation score for a particular user (e.g., Alice)is as follows. Each time Alice diggs a story in the upcoming story poolthat is subsequently promoted, Alice's user reputation score isincremented by a fixed value. Each time Alice burries a story in theupcoming story pool that is subsequently removed from the pool, Alice'suser reputation score is also incremented by a fixed value. In someembodiments multiple user reputation scores are maintained for Alice,e.g., with one relating to her success at digging stories and anotherrelating to her success at burying stories. Alice may also havedifferent user reputation scores for each category or type of content,for example, to reflect that she is astute at selecting sports relatedstories for promotion, but not very good at selecting photographs thatare ultimately promoted.

As another example, whenever a story is promoted, each of the users thatdugg that story (also referred to herein as having “voted” for thestory) prior to the story being promoted receives a fractional share ofthe fixed value, added to their respective user reputation scores. Thescore of the original submitter is, in some embodiments increased by ahigher fixed value than those that digg the story—in other embodimentsthe submitter is treated as being the first digger of the story. Asanother example, the first n users who digg a story, or any users whodigg the story within n amount of time from when the story is submitted,have their user reputation scores incremented while later digging usersdo not, irrespective of whether the later digging users dugg the storyprior to its promotion. The scores may be updated by an equal value, ormay take into account the timing of the diggs. For example, the firstperson to digg a story that is ultimately promoted may receive a higherincrease to his user reputation score than the fifth digger, who alsoreceives an increase to his score. In various embodiments, only storiesdugg in the last 30 days or some other time are considered whendetermining user reputation scores. Additional techniques fordetermining a user reputation score are described in more detail below.

At 1606, a content reputation score is determined for each item in thegroup of candidate content items. One way of calculating a contentreputation score is to sum the user reputation scores of each of theusers that voted (or dugg) the story. In various embodiments, the userreputation scores of some users is not considered at 1606. For example,users that have recently registered may not have their user reputationscores be used in any computations until two weeks have elapsed.

At 1608, a determination is made as to whether or not the content shouldbe promoted. For example, at 1608 the content reputation score for eachcandidate content item is compared against a threshold. If the thresholdis exceeded, the content is promoted. If some of the content reputationscore is determined to include cohort digging, the threshold is raised,or the score attributable to the cohort digging activity is discarded,as applicable. In various embodiments, additional checks and/orthrottling are performed at 1608. For example, promotion engine 118 maybe configured to perform the processing shown in FIG. 16 as a regularlyoccurring batch process. For any particular iteration of the process, alimit may be imposed by promotion engine 118 on the number and/or natureof content that can be promoted. For example, promotion engine 118 maybe configured with a rule that only five images may be promoted perexecution of the process, that a maximum of one story per category maybe promoted, or that a total of ten items may be promoted per cycle,across all content.

In some embodiments content items that are otherwise promotable (e.g.,that have content reputation scores exceeding the appropriate threshold)are sorted based on the ratio between their respective contentreputation scores and the applicable threshold values. The ratio (e.g.,the sorted list) is used as tie breaker information in the event thatmore content is promotable than the rules allow to be promoted. In someembodiments, promotable items that are not promoted are considered forpromotion on the next iteration of the process shown in FIG. 16. Inother embodiments, promotable items are placed in a queue and releasedfrom the queue after a period of time has elapsed. In some embodimentsall promotable items are placed in a queue prior to promotion so that anadministrator has time to review the content for violations of serviceterms or other reasons.

In some embodiments different thresholds are used for determiningwhether to promote content in different categories and/or content ofdifferent types. For example, a higher threshold may be used for thepromotion of sports news than for the promotion of astronomy stories.Additionally, multiple thresholds can be employed when determiningwhether to promote a particular content item, such as requiring that atotal digg count threshold be exceeded for the story, along with thecontent reputation threshold. In the event that a total digg count isconsidered, in some embodiments the total digg count for the story isevaluated at 1602 as part of the initial candidate selection.

Yet another technique for calculating a user reputation score is asfollows. Let u be a user, Iu be the set of content dugg by the user inthe last 28 days, Pu be the subset of items in Iu that were promoted andfor which the user was an “early” digger, and Di be the current diggtotal of item i. The user reputation score (su for the user u can beexpressed as:

$s_{u} = {\frac{\sum\limits_{i \in I_{u}}D_{i}}{{I_{u}} + 30}.}$

The number 30 in the denominator is a regulator that compensates forissues such as when a new or inactive user diggs an item whichsubsequently becomes very popular. It regulates exceptionally high userreputation scores that could otherwise result. In various embodiments,the total number of diggs made by a user are considered when determininga user reputation score so that an indiscriminate digger cannot achievea high user reputation score by merely digging every single story in theupcoming pool.

Automatically Adapting Thresholds

In some embodiments, promotional requirements (e.g., the amount of thecontent reputation score and/or total digg count) are dynamicallyadapted and the corresponding thresholds are raised and lowered based ona variety of factors. Thresholds can be automatically recalculated,e.g., once a day, once a week, once a month, based on historicalinformation. Examples of factors include:

1. An assessment of diversity of digging in a given topic. Diversity canbe a measurement of many things. One example is the location of thedigging activity within the site, such as on a permalink page vs. fromusers' friends profiles. Other diversity examples include same-IP andother cohort-related information. Diversity can be required so that ifsomething is insufficiently (sufficiently) diverse, the bar forpromotion for a given story can be raised (or lowered) accordingly.

2. The velocity at which diggs are received for the content.

3. The total number of visitors to a given topic (e.g., based on a trendof historical data in the topic and/or relative to other topics). Forexample, during annual sporting events, during elections, etc., a surgeof interest in a topic should typically require an increase in therequirements for a story associated with that event to be promoted. Atother times of the year, considerably fewer visitors are likely toreview content associated with the topic and the threshold will adjustdownward accordingly.

4. Activity during specific times of day (e.g., requiring fewer diggs at3 am Pacific on a Saturday/Holiday than at 11 a Pacific on a Monday).This can also be based on peaks and valleys in site traffic, rather thanor in addition to precise times of day (e.g., clock values).

5. The likelihood of a story being promoted given the submitter'shistory and the distribution and reach of his/her friend network.

6. The number of burries against the content. A story with 25 diggs and1 bury is promoted, while a story with 25 diggs and 3 buries requiresmore diggs to be promoted, e.g., exponentially based on the number ofburries.

7. If the site has been repeatedly submitted by the same user whosubmits nothing else.

8. Sophistication of the “average” user interested in the topic. On wayof evaluating this factor is as follows. First, “new” users (e.g., thoseactive less than two weeks) are removed. A determination of userreputation scores of all of the remaining users is made and an averageis taken. If, over time, the population becomes more sophisticated(higher average), a higher content reputation can be required to promotea story. If the population becomes less sophisticated (e.g., due to aschool session starting and many users having joined the system 30 daysearlier), a lower content reputation can be required to promote a story.

Thresholds may also be applied selectively to, e.g., differentcategories, such that Astronomy uses a fixed set of promotion thresholdswhile Operating Systems is dynamic. Events such as the occurrence ofmajor news events (e.g., an earthquake) can also be used to trigger ordisengage the application of dynamic thresholds to categories.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method of determining whether to promote afirst content item, with consideration of orchestrated activityregarding the first content item, the method comprising: receiving,regarding the first content item, a plurality of new preference eventsfrom a plurality of users; identifying a first group of cohorts thatpreviously orchestrated preference events; identifying a subset of theplurality of users as members of the first group of cohorts, wherein thecohorts are users that initiated the same preference events for multiplecontent items other than the first content item; determining whether thesubset of the plurality of users is of at least a threshold size; and ifthe subset of the plurality of users is of at least the threshold size,determining whether to promote the first content item, wherein promotingthe content item comprises increasing a prominence of the first contentitem in an interface through which content items are presented to users.2. The method of claim 1, wherein a preference event is one of a digg, abury and a comment.
 3. The method of claim 1, wherein the multiplecontent items were submitted by the same user that submitted the contentitem.
 4. The method of claim 1, wherein identifying users as members ofthe first group of cohorts comprises correlating activities of theplurality of users.
 5. The method of claim 1, wherein identifying usersas members of the first group of cohorts comprises: retrieving a list ofknown groups of associated users; and searching the known groups for theplurality of users.
 6. The method of claim 5, wherein identifying asubset of the plurality of users as members of the first group ofcohorts comprises: determining whether the same preference event wasreceived from a threshold number of associated users in one group ofassociated users.
 7. The method of claim 1, wherein identifying users asmembers of the first group of cohorts comprises: maintaining a table ofknown cohorts; and periodically updating the table as additionalpreference events are received.
 8. The method of claim 1, whereinidentifying a subset of the plurality of users as members of the firstgroup of cohorts comprises: within the plurality of users, identifying aset of users that submitted the same preference event regarding thefirst content item; and determining whether the set of users submittedidentical preference events for a threshold number of other contentitems.
 9. The method of claim 1, wherein identifying a subset of theplurality of users as members of the first group of cohorts comprises:detecting associated events among the received preference events. 10.The method of claim 9, wherein detecting associated events comprisescorrelating behavior among the plurality of users.
 11. The method ofclaim 1, wherein determining whether to promote the first content itemcomprises: increasing a score associated with the first content item ifit is determined that: the threshold number of cohorts initiatednegative preference events for the first content item.
 12. The method ofclaim 1, wherein determining whether to promote the first content itemcomprises: decreasing a score associated with the first content item ifit is determined that: the threshold number of cohorts initiatedpositive preference events for the first content item.
 13. The method ofclaim 1, wherein determining whether to promote the first content itemcomprises: discarding preference events initiated by the subset of theplurality of users.
 14. A system for determining whether to promote afirst content item, with consideration of orchestrated activityregarding the first content item, the system including: a processor; anda memory coupled with the processor, wherein the memory is configured toprovide the processor with instructions that, when executed, cause theprocessor to: receive a plurality of new preference events regarding thefirst content item, from a plurality of users; identify a first group ofcohorts that previously orchestrated preference events; identify asubset of the plurality of users as members of the first group ofcohorts, wherein the cohorts are users that initiated the samepreference events for multiple content items other than the firstcontent item; determine whether the subset of the plurality of users isof at least a threshold size; and if the subset of the plurality ofusers is of at least the threshold size, determine whether to promotethe first content item, wherein promoting the first content itemcomprises increasing a prominence of the first content item in aninterface through which content items are presented to users.
 15. Thesystem of claim 14, wherein the preference event is one of a digg, abury, and a comment.
 16. The system of claim 14, wherein identifyingusers as members of the first group of cohorts comprises correlatingactivities of the plurality of users.
 17. The system of claim 14,wherein determining whether to promote the first content item comprises:increasing a score associated with the first content item if it isdetermined that: the threshold number of cohorts initiated negativepreference events for the first content item.
 18. The system of claim14, wherein determining whether to promote the first content itemcomprises: decreasing a score associated with the first content item ifit is determined that: the threshold number of cohorts initiatedpositive preference events for the first content item.
 19. The system ofclaim 14, wherein determining whether to promote the first content itemcomprises: discarding preference events initiated by the subset of theplurality of users.
 20. A non-transitory computer-readable mediumstoring instructions that, when executed by a processor, cause theprocessor to perform a method of determining whether to promote a firstcontent item, the method comprising: receiving, regarding the firstcontent item, a plurality of new preference events from a plurality ofusers; identifying a first group of cohorts that previously orchestratedpreference events; identifying a subset of the plurality of users asmembers of the first group of cohorts, wherein the cohorts are usersthat initiated the same preference events for multiple content itemsother than the first content item; determining whether the subset of theplurality of users is of at least a threshold size; and if the subset ofthe plurality of users is of at least the threshold size, determiningwhether to promote the first content item, wherein promoting the contentitem comprises increasing a prominence of the first content item in aninterface through which content items are presented to users.
 21. Thesystem of claim 14, wherein determining whether to promote the firstcontent item comprises identifying an effect of the preference eventsinitiated by the subset of the plurality of users on a score of thefirst content item.
 22. The system of claim 21, wherein determiningwhether to promote the first content item further comprises removing theeffect of the preference events initiated by the subset of the pluralityof users on the score of the first content item.
 23. The method of claim1, wherein determining whether to promote the first content itemcomprises identifying an effect of the preference events initiated bythe subset of the plurality of users on a score of the first contentitem.
 24. The method of claim 23, wherein determining whether to promotethe first content item further comprises removing the effect of thepreference events initiated by the subset of the plurality of users onthe score of the first content item.